If you are looking to define a new piece via the GUI, that would be a heavy lift. I sincerely doubt anybody has made that possible. In Popeye you could (in principle) add a new piece in the c code. Instead I suggest you check out the FairyMax engine, which uses a file fmax.ini to define the variants and the piece movements. It's still complicated, but if your piece combinations are simple you might be able to figure it out. For example look at the Queen, look at the Knight, look at the Amazon, and see in what way the Queen and Knight were combined there. Then try the same thing with your desired piece combination.

For resizing the board, both Xboard(Winboard) and Popeye can do it. In Xboard you can change the size of the board _larger_ on the fly by providing a larger FEN. But if you try to make it smaller with the same trick then it will complain. There are command-line options as well which you can read up on. With Popeye you can change the size of the board by editing board.h and compiling. In board.c the defined files go from a to z, so it should probably handle up to a 26x26 board this way.
 
Lines in board.h to change:

  nr_files_on_board = 8,
  nr_rows_on_board = 8,